Introducing re-weighted range voting in clinical practice guideline prioritization: Development and testing of the re-weighted priority-setting (REPS) tool

We aimed to develop and test a tool based on the re-weighted range voting mechanism to prioritize items (i.e. key questions) in a priority-setting assessment for clinical practice guidelines. The secondary aim was to provide methodological context of the tool. We iteratively developed the tool and used qualitative methods (i.e. think-aloud and semi-structured interviews) to test the tool’s usability and make adjustments accordingly. An observational approach was used to test the tool’s outcome satisfaction in a real-world priority-setting assessment within a rare-disease guideline of a European Reference Network and under four different conditions in the tool. Four guideline methodologists tested the usability of the tool. The real-world testing was performed with a guideline panel consisting of a core working group, five expertise working groups, and a working group with patient representatives. Thirty-one panel members assigned scores in the priority-setting assessment. Seventeen panel members rated the priority-setting outcome, and sixteen panel members rated the outputs generated under the four conditions. Upon initial use, guideline methodologists found the tool to be quite overwhelming. However, with some initial effort they were able to easily identify the tool’s structure. Based on observations and feedback, the tool was further refined and user guidance was developed. Guideline panel members expressed (high) satisfaction with the priority-setting outcome. They particularly preferred the condition when using mean subgroup scores as input or employing aggressive penalties in the weighting method to determine the outputs. The tool generates a ranked list of items and offers flexibility for different choices in priority-setting assessments as long as its input format requirements are met. Although it is not a consensus method, the tool assists in narrowing down a set of priority items. Additional steps in the priority-setting assessment can lead to a consensus being reached regarding the final outcome.

Note: the REPS-tool is currently programmed in Microsoft Excel and uses macros.Make sure that Microsoft Excel allows macros on your computer, otherwise the REPStool cannot be used.Microsoft Excel might also automatically block macros in Excelfiles downloaded from the internet.To use such Excel-files, it is necessary to check the 'unblock' checkbox in the file properties of the Excel-file.To learn more, see: https://learn.microsoft.com/en-gb/DeployOffice/security/internet-macros-blocked

Purpose of the REPS-tool
The main purpose of the REPS-tool is to aid in the priority-setting of any item (e.g.clinical practice guideline, systematic review, recommendation, key question, etc.) based on priority scores assigned by participants in the priority-setting assessment by providing a ranked list of items as its output.It aims to be a flexible tool that allows for all different kinds of procedures in the priority-setting assessment before and after using the tool, as long as the input complies to the input format requirement.
Two different types of weighting methods are available [1, 2]: -The regular Re-weighted Range Voting method -The decay-adjusted weighting method, based on the regular weighting method, where the decay pattern of individual weights can be adjusted

Role in the priority-setting assessment
The REPS-tool is a function component in a priority-setting assessment.The function needs input according to a specific input format and produces an output in the form of a ranked list.The function's mechanism is re-weighted range voting to assign ranks.
All steps prior to using the function can be considered procedural steps ultimately leading to the correct input format for the function.Organization may wish to use their own set of selected priority indicators, may wish to ask participants to score multiple indicators per item and use the mean score, may wish to use mean scores of groups of stakeholders, and/or any other preferred procedural step.However, it should lead to data satisfying the input format for the REPS-tool.
Once data is entered in the REPS-tool and ranks are assigned, the tool's output can immediately be used as the outcome of the priority-setting assessment (Fig. 1).However, organizations may wish to add one or several procedural steps after the function component.For example, the tool's ranked top 15 output could be used to discuss and form a top 5 priority items as a definitive outcome of the priority-setting assessment.
Note: the REPS-tool is not a consensus method.It ranks items based on priority scores assigned by a group of participants.Though, consensus might later be achieved using the tool's output by additional steps in the priority-setting assessment when desirable.

Re-weighted range voting introduction
The REPS-tool uses re-weighted range voting as a mechanism to form a list of items ordered ascendingly by their assigned ranks.The ranks are assigned based on the highest sum score of the items.However, after every assignment of a rank to an item, a weight for each of the participants in the priority-setting assessment is being (re)calculated based on the scores that the participant already assigned to the previously ranked item(s).All participants have a weight starting at '1'.The higher the scores on the previous ranked items for each participant, the lower the individual weight when recalculated.The lower the weight, the less influence on the next item to be ranked as the individual weights are multiplied by the participant's item scores.The re-weighted participants scores are summed to form new item sun scores and the highest item sum score is assigned the subsequent rank.
The calculation of the individual weights is, among others, based on the maximum scale score.Every participant scores items on a scale from 0 to a maximum predefined score (e.g. 5).There is no limit for items to receive the same priority score, so, in theory, a participant could rate all items with the same priority score.
The scale uses a distinctive description.It is not allowed to assign negative priority scores (e.g.-1).The scores can be interpreted as follows: 0 = No priority 1 = Lowest priority possible … Maximum scale score = Highest priority possible The REPS-tool will not adjust the individual weight when a participant assigned a score of 0 or refrained from assigning a score to a winning item.The REPS-tool consists of three worksheets: RRV (Fig. 2), Labels list (Fig. 3), and Ranking outcome (Fig. 4).
Entry of participant data and priority scores, heterogeneity analyses, and ranking is conducted in the 'RRV' worksheet.It contains six sections: item labels, ranking information, heterogeneity analysis, tool parameters, participants and scores, and ranking buttons.

Section A (Fig. 2A) -Item labels
Item names cannot be filled out in section A in the RRV work sheet.Rather, they are filled out under the Labels list worksheet and projected in section A (Fig. 2A).Item names of items to be prioritized (e.g.CPG names, section titles, recommendations, key questions, etc.) can be placed in the 'Labels list' worksheet (Fig. 3).If a list of item names is available, then the list can be copy/pasted into the Labels list worksheet (Fig. 3).

Figure 2 -Overview of the 'RRV' worksheet in the REPS-tool opened in Microsoft
Excel.

Section B (Fig. 2B) -Ranking information
Here, (un)weigted sum scores can be seen per item in each respective column for informative reasons.A winner is automatically detected by the tool, showing a green cell with 'WINNER' in the winning item's column.Although assigning ranks to winning items can be performed semiautomatically, a rank can also be assigned manually (e.g. in case of a tie) by inserting a number in the item's cell on the row named 'Rank' (i.e.immediately above the green cell indicating 'WINNER').

Section C (Fig. 2C) -Heterogeneity analysis
This section concerns heterogeneity analyses.For each item some measures of central tendency and measures of dispersion are displayed in the item's respective column.

Section D (Fig. 2D) -Tool parameters
Tool parameters can be adjusted here.Available tool parameters are the constant, the maximum scale score, the weighting method, and the decay aggression (only available for the decay-adjusted weighting method).Changing these parameters will not result in the tool providing any direct feedback, but rather different formulas will operate in the background.

Section E (Fig. 2E) -Participants and scores
The columns indicating the 'individual weight' and 'sum of picks' are informative for the user and are part of the calculations in the background.The column 'organization' and 'name' are optional, but can be used to identify the participant.Importantly, the field with 'scores' align with the items (columns) and participants (rows) and therefore forms a matrix.

Section F (Fig. 2F) -Ranking buttons
This section contains two buttons: one to assign a rank to the identified winner and one to clear all assigned rankings.Items with assigned ranks are displayed in the Ranking outcome worksheet (Fig. 4).This list can be copy/pasted to word-processing software and is, effectively, the output of the REPS-tool.

Input format
From Fig. 2A and Fig. 2E it could already be deduced that the columns represent the items to be prioritized and that the rows represent participants.The input format for the tool is thus a matrix with priority scores corresponding to an item and a participant: Any priority-setting process may precede, as long as the eventual data used for the REPS-tool complies to the input format in Matrix 1.For example, the scores in Matrix 2 (hypothetical Note: Worksheets can be protected (whether or not by password) so that cells and formulas are locked in their respective cells when desirable.Restricting cells may also result in users not able to select irrelevant cells guiding their experience in using the REPS-tool.
example) are overall scores the participants assigned based on four priority indicators.However, it is also possible to provide the mean or median of separately scored priority indicators per item as input for the REPS-tool.Nonetheless, the options in the score scale should always include 0 (i.e.no priority), as a score of 0 will not adjust the individual weight.
Furthermore, when there are significant uneven numbers of delegates from participating stakeholders and this is considered undesirable, the mean or median of the participating delegates of stakeholders can be used as an input in the REPS-tool.Rows then become the stakeholder organizations or (sub-)groups instead of individuals (Matrix 3).

Obtaining priority scores
It is important to recognize that the REPS-tool uses a single score or no score (when participants refrained from assigning a score) per item.Thus, either the cell contains a number or is left blank.There are several ways to obtain a single score, for example: -Ask to assign an overall priority score for each item -Ask to assign a priority score on a single priority indicator for each item -Ask to assign priority scores on multiple priority indicators for each item and average the assigned scores on the priority indicators for each item.
Matrix 3 -Input format for the REPS-tool when using, for example, the mean of participating delegates per organization.Organizations may have different needs and may operate in different contexts.It might be helpful to select priority indicators which seem relevant to the context and needs.For example, priority indicators might be dependent of the field the priority-assessment takes place, or for which purpose the item is prioritized for (e.g.update, development de novo, implementation), or even the health care system the organization is operating in.
Scores can be elicited by sending out an (online) survey, for example.In the future, prioritysetting tools may be programmed into applications from where the scores might also be elicited.

Data entry
The item names in Matrix 2 are hypothetical examples of questions prioritized to be developed de novo.Item names are automatically displayed in the corresponding columns in the 'RRV' worksheet when the names are placed in the list in the 'Labels list' worksheet (Fig. 5).
Organization, participant names and scores are easily copied and pasted in the tool when using a dataset structured according to the input format.Names and priority scores from Matrix 2 were pasted in the REPS-tool (see Fig. 6).The item names can also be copy/pasted in the 'Labels list' worksheet.It is advisable not to cut/paste or move cells within the tool as underlying cell references may become dislocated, even when the worksheet is protected.The columns 'Name' and 'Organization' do not necessarily have to contain data to perform the ranking of items in the REPS-tool.However, it is suggested to at least fill in the participant's name (or identifier) when later identification is desirable (e.g. to check for errors in specific rows of data).

Note
Figure 5 -Item names placed in the 'Labels list' worksheet (Fig. 2) are automatically displayed in the 'RRV' worksheet (Fig. 1A) Note: Using the same order of participants and items from the dataset in the REPStool (i.e. in the 'RRV' and 'Labels list' worksheets) ensures that priority scores can be pasted in the tool without data entry errors.The item names may be horizontally oriented in the dataset.The transpose function Microsoft Excel (e.g. in a new workbook) can be used to create a vertically oriented list following the order of items used in the dataset.The vertical list can be pasted in the 'Labels list' worksheet and item names will appear automatically in the corresponding columns in the 'RRV' worksheet (Fig. 1A).In this section the scores can be validated as well.That is, cells with input can be highlighted in color that contain text, values smaller than 0, or values larger than the maximum scale score.Clicking on the button 'Validate scores' will highlight the cells (Fig. 7).The button 'Clear validation' will remove the highlights.

Weighting methods and parameters
The REPS-tool has two weighting methods for individual weights: the regular re-weighted range voting method (Formula 1) and the decay-adjusted weighting method (Formula 2) [1, 2].The latter formula can be used for disproportionate representation [2].
The regular re-weighted range voting method will result in a proportional representation (as assigned with priority scores) reflected in the ranking [1,2].The decay-adjusted weight (with A>1) results in a more aggressive decay of the individual weights (Fig. 8).A relative aggressive decay (e.g.A=4) causes a steep decline of the individual weight immediately with the first few points assigned to a winning item.This could theoretically leave room for boosting less represented perspectives in the ranking outcome, as participants 'having their way' with the first few winners will have a very low individual weight thereafter.
(1) RE-weighted Priority-Setting (REPS) A quick-start guide to the REPS-tool The weighting parameters of the individual weights can be selected in the weighting parameters section in the 'RRV' worksheet of the REPS-tool (Fig. 2D, Fig. 9).The value in the cell right to 'Max score:' indicates the maximum scale score.That is, the maximum score a participant is allowed to assign to an item.A brief explanation is provided on the right of the value in the REPStool which adapts according to the input value of the maximum scale score.For example, using a maximum scale score of 7, the participants essentially score on an 8-item scale ranging from 0 to 7.
The formulas to calculate individual weights need a constant in both the numerator and denominator in order to result in a weight of '1' when no items were ranked or when no scores were assigned to the previous ranked item(s).The constant can be any positive number [1].
Figure 8 -A graph showing the decay of individual weights using the decay-adjusted weighting method (Formula 2), a constant of K=0.5, and a maximum scale score of 10.The regular re-weighted range voting method (Formula 1) is identical to A=1.By changing the value of the cell right to 'Weighting method:', the weighting method can be changed.The regular re-weighted range voting method (Formula 1) is used when the value is '0', while the decay-adjusted method (Formula 2) is used when the value is '1'.The value next to the cell 'Decay aggression (A):' is the value of exponent A in Formula 2.

Assigning ranks
The REPS-tool immediately indicates the first 'winner' in green when scores are entered (Fig. 10).Make sure not to assign ranks until all data required following the input format is entered to prevent a faulty ranking.In our hypothetical example from Matrix 2, the first winner is "What is the role of biopsy in the detection of a hepatocellular carcinoma?"because it had the highest sum score (i.e.13).
By clicking on the 'Next ranking ->' button (Fig. 11), the identified 'winner' receives its rank and the next 'winner' is identified to receive a rank.The button enables a semi-automatic ranking of items (Fig. 12).A message will appear to inform the user there is no item left to rank when the button is clicked and there are no more items left.Ranks can also be manually entered in the row 'Rank:' (e.g.fill out '1' in the cell above the cell indicating the winner in Fig. 10).After accepting the value in the cell, the REPS-tool will immediately identify the next 'winner'.
Figure 10 -Automatic indication of the winner in the ranking information section (also see Fig. 2B).A rank is not yet assigned to the winner at this point.

Note:
The tool does not provide immediate feedback when changing parameters.It switches formulas in the background to calculate the individual weights, for example when switching between 0 and 1 as the weighting method.
The row 'Rank:', containing all of the assigned ranks to the items in the corresponding columns (Fig. 10), can be cleared at once by clicking on the 'Clear ranking' button (Fig. 11).
In some cases, the (new) sum scores of multiple items may be equal and these items are tied for a rank.Equal sum scores are most likely to happen during the first few assignments of ranks, whereafter it probably becomes less likely that items have exactly the same sum score due to the individual weighting of the assigned scores.
The REPS-tool will indicate which items have identical scores with 'MULTI (n)' in red when tied for a rank (Fig. 13).Here, n displays the number of items with identical scores.Clicking the 'Next ranking ->' button when multi-winners are identified will not assign a rank to any of these items.The 'Next ranking ->' button, however, prompts a message indicating for which rank the items are tied.This rank needs to be manually assigned to one of the multi-winner items.There are no prespecified rules for this decision yet, however heterogeneity analysis may aid in the Figure 12 -Semi-automatic ranking using the 'Rank next ->' button (Fig. 11) until there are no items left to rank, which prompts a message to inform the user.
decision.For example, one might choose an item where there is some agreement among the participants reflected by a smaller variance, standard deviation, or range compared to the other multi-winner item(s).
All items who received a rank will be displayed in the Ranking outcome worksheet ordered by their rank.When deciding to change the tool's parameters after a ranking is completed, the tool does not refresh the assigned ranks automatically.Thus, after changing the tool's parameters, the ranking should be cleared before new ranks are assigned according to the new parameters.

Heterogeneity analyses
The REPS-tool has a section for heterogeneity analyses (Fig. 2C).Analyses are carried out on the unweighted scores assigned by the participants.Central tendency measures (i.e.mean and median) are provided, accompanied by measures of dispersion (i.e.variance, standard deviation, range, and interquartile range) and sizes of the quartiles.2) in the tool.The REPS-tool identifies which items are tied by displaying MULTI(n) in the item column and shows how many items are tied within the parentheses.Clicking the 'Rank next ->' button (Fig. 11) will prompt a message showing for which rank the items are tied.In this case there is a tie for rank 2.
Note: Do not assign identical ranks to multiple items.These items will not be displayed correctly in the 'Ranking outcome' worksheet.For example, do not manually assign the same rank to the items identified as a multi-winner.
Items meeting specific conditions, such as a variance greater or equal to three, can be highlighted in the REPS-tool (Fig. 14).Fill out and accept a value in the yellow (smaller or equal to) or red (greater or equal to) cell right to 'Highlight when:' for the tool to automatically highlight items meeting the condition.
When items are tied for a rank and the rank has to be entered manually, heterogeneity analyses may be of use in deciding which item receives the rank.Although there is currently no consensus and there are no predefined decision-rules, the measures of dispersion might indicate the level agreement among participants.For example, large variance could mean that the participants do not agree about the priority while little variance could indicate that there is more agreement about the level of priority.One might choose, for example, to assign the rank to the item with the smallest variance.

Tool output
The output of the tool is a list of items ascendingly ordered by the assigned ranks to the items.Only items that were (manually or semi-automatically) assigned a rank to in the 'RRV' worksheet are displayed in the 'Ranking outcome' worksheet of the REPS-tool (Fig. 15).It is possible to create a top 10 or a top 15, by (manually or semi-automatically) assigning a rank to the first 10 or 15 winners, respectively.That means that solely items that had a rank assigned to them in the 'RRV' worksheet are ordered and displayed in list in the 'Ranking outcome' worksheet.The list can be copied from the worksheet to other applications, such as Microsoft Word.

Adapting the REPS-tool
The formulas in the worksheets and the worksheets themselves are not locked in the REPS-tool.Therefore, the tool can be programmed in Microsoft Excel according to specific needs not currently programmed in the tool.For example, it is conceivable to wish for a different weighting of individual weights for patients participating in the priority-setting assessment.Or, a different set of weighting methods could be programmed than currently available in the tool.Although these aspects are not features in the current version of the REPS-tool, it should be programmable using Microsoft Excel when desirable.The REPS-tool may be further edited by programming specific wishes for an organization in the tool.In fact, we would encourage any adaptations that would allow for its use for different contexts and needs.This includes prioritysetting outside the context of clinical practice guidelines.

Figure 1 -
Figure 1 -A process component leading to the input of the function component.The output of the function component is immediately used as the outcome of the prioritysetting assessment, although the latter may not always desirable.

Figure 3 -
Figure 3 -Detail of the 'Labels list' worksheet in the REPS-tool opened in Microsoft Excel.

Matrix 1 -Figure 4 -
Figure 4 -Detail of the 'Ranking outcome' worksheet in the REPS-tool opened with Microsoft Excel.Excel shows #N/B because none of the items were currently ranked.

Figure 6 -
Figure 6 -Names and scores were entered into the REPS-tool ('RRV' worksheet)

Figure 7 -
Figure 7 -With a maximum scale score of '3' the 'Validate scores' button will highlight text, any score above 3, and any score below 0. The 'Clear validation' button will remove all highlights.

Figure 11 -
Figure 11 -Buttons for the semi-automatic assignment of ranks and for clearing all ranks.

Figure 13 -
Figure 13 -Two items are tied for a rank as indicated by MULTI(2) in the tool.The REPS-tool identifies which items are tied by displaying MULTI(n) in the item column and shows how many items are tied within the parentheses.Clicking the 'Rank next ->' button (Fig.11) will prompt a message showing for which rank the items are tied.In this case there is a tie for rank 2.

Figure 14 -
Figure 14 -Highlight items for central tendency and dispersion measure values 'smaller than or equal to' (yellow) and 'greater than or equal to' (red).Items meeting the condition are highlighted in their respective column.

Figure 15 -
Figure 15 -The list of items that received a ranking ordered by rank in the 'Ranking outcome' worksheet.Microsoft Excel shows #N/B because a fifth item was not assigned a rank.

:
Literature reviews reporting priority indicators for guidelines are being published in scientific journals.Guideline developing organizations may use such overviews to select priority indicators relevant for their priority-setting assessment.